White balance adjusting apparatus, operation method thereof, and non-transitory computer readable medium

ABSTRACT

A non-emission image is obtained by a non-emission image obtaining unit in a state in which a plurality of flash devices do not emit light. Pre-emission images are obtained by an emission image obtaining unit in a state in which the plurality of flash devices individually emits light. Flash light irradiation areas are specified by a flash light irradiation area specifying unit based on a signal value difference of a plurality of division areas of the non-emission image and each of the emission images. A priority flash device selecting unit selects a priority flash device as a target of white balance (WB) adjustment. A WB adjusting unit performs WB adjustment based on signal values of irradiation areas of the selected priority flash device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of PCT International Application No.PCT/JP2017/006233 filed on 20 Feb. 2017, which claims priority under 35U.S.C § 119(a) to Japanese Patent Application No. 2016-073268 filed on31 Mar. 2016. The above application is hereby expressly incorporated byreference, in its entirety, into the present application.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a white balance adjusting apparatus, anoperation method thereof, and a non-transitory computer readable mediumwhich adjust white balance at the time of imaging using a plurality ofauxiliary light sources.

2. Description of the Related Art

Human visual perception has color constancy. Accordingly, it is possibleto perceive an original color of a subject irrespective of a differenceof ambient light such as electric light, fluorescent light, or sunlight.In contrast, an image using an imaging device such as a digital camerais directly influenced by the ambient light. Thus, the imaging devicehas a white balance adjusting function of performing color conversion onthe image such that the human can see a natural image by correcting theinfluence of the ambient light.

For example, a main subject is irradiated with mixed light of theambient light with flash light on the image captured by the imagingdevice by using a flash device as an auxiliary light source. Abackground is less influenced by the flash light, and is mostlyirradiated with the ambient light.

For example, in auto white balance adjustment at the time of generalflash imaging, a ratio of the ambient light to the flash light(hereinafter, referred to as a mixed light ratio is calculated, andwhite balance is adjusted according to the mixed light ratio, asdescribed in JP2010-193048A. There is a tendency to irradiate the mainsubject with the flash light at the time of single flash imaging usingone flash. Thus, the main subject has an appropriate tint by performingthe auto white balance adjustment according to the mixed light ratio ofa portion irradiated with the flash light.

SUMMARY OF THE INVENTION

However, in the imaging using a plurality of auxiliary light sources,for example, a plurality of flash devices, the portion stronglyirradiated with the flash light may not be the main subject. Forexample, in a case where there is the plurality of auxiliary lightsources such as the flash device that irradiates the main subject withthe flash light and the flash device that irradiates the background withthe flash light, the flash device that irradiates the background maystrongly emit the light. In this case, in a case where the auto whitebalance adjustment is performed depending on the mixed light ratio inthe portion strongly irradiated with the flash light, the image has atint that emphasizes the background, and thus, the tint of the mainsubject becomes bad.

The present invention has been made in view of the circumstances, and anobject of the present invention is to provide a white balance adjustingapparatus, an operation method thereof, and a non-transitory computerreadable medium which allow a main subject to have an appropriate tintat the time of imaging using a plurality of auxiliary light sources.

In order to achieve the object, a white balance adjusting apparatus ofthe present invention comprises a non-emission image obtaining unit, anemission image obtaining unit, an auxiliary light irradiation areaspecifying unit, a priority auxiliary light source selecting unit, awhite balance adjustment value calculating unit, and a white balanceadjusting unit. The non-emission image obtaining unit obtains anon-emission image by imaging a subject in a state in which a pluralityof auxiliary light sources does not emit light. The emission imageobtaining unit obtains emission images of the auxiliary light sources byimaging the subject in a state in which the plurality of auxiliary lightsources individually emits light. The auxiliary light irradiation areaspecifying unit divides the non-emission image and each of the emissionimages into a plurality of division areas, and specifies auxiliary lightirradiation areas irradiated with auxiliary light of each of theauxiliary light sources based on a signal value difference of eachdivision area between the state in which the plurality of auxiliarylight sources individually emits light and the state in which theplurality of auxiliary light sources does not emit light. The priorityauxiliary light source selecting unit selects a priority auxiliary lightsource as a target of white balance adjustment from the auxiliary lightsources. The white balance adjustment value calculating unit calculatesa white balance adjustment value based on signal values of priorityauxiliary light irradiation areas irradiated with auxiliary light of theselected priority auxiliary light source. The white balance adjustingunit performs adjustment using the white balance adjustment value.

It is preferable that the priority auxiliary light source selecting unitincludes a face area detecting unit and a priority auxiliary lightsource determining unit. The face area detecting unit detects face areasfrom the non-emission image or the emission images. The priorityauxiliary light source determining unit specifies which of the auxiliarylight irradiation areas the face areas detected by the face areadetecting unit are present, and determines that the auxiliary lightsource corresponding to the auxiliary light irradiation areas includingthe face areas is the priority auxiliary light source.

The priority auxiliary light source selecting unit includes a priorityauxiliary light source determining unit that determines the priorityauxiliary light source based on previously stored light source colorinformation of the auxiliary light source. The priority auxiliary lightsource determining unit sets a determination range in a color space byusing the previously stored light source color information of theauxiliary light, light source color information of ambient lightobtained from the non-emission image, and pixel information at the timeof non-emission of the auxiliary light irradiation areas. The priorityauxiliary light source determining unit determines the auxiliary lightsource corresponding to the auxiliary light irradiation areas as thepriority auxiliary light source in a case where the pixel informationbased on the emission image is positioned within the determinationrange.

It is preferable that the priority auxiliary light source is determinedbased on the non-emission signal value average, the signal value averageprediction value at the time of emission of the auxiliary light source,and the emission signal value average. The light source colorinformation of the auxiliary light is coordinates indicating a color ofthe auxiliary light in a color space. The light source color informationof the ambient light is coordinates which are obtained based on thenon-emission image and indicate a color of the ambient light in thecolor space. The pixel information at the time of the non-emission ofthe auxiliary light irradiation areas is coordinates which are obtainedbased on the non-emission image and indicate a non-emission signal valueaverage of the auxiliary light irradiation areas in the color space. Thepriority auxiliary light source determining unit calculates the emissionsignal value average which is the signal value average of the auxiliarylight irradiation areas in the color space based on the emission image.The priority auxiliary light source determining unit calculates adifference vector which is a difference between the light source colorinformation of the auxiliary light and the light source colorinformation of the ambient light, and obtains the signal value averageprediction value at the time of the emission of the auxiliary lightsource by adding the difference vector to the coordinates of thenon-emission signal value average.

It is preferable that the priority auxiliary light source determiningunit determines that the auxiliary light source is the priorityauxiliary light source in a case where the emission signal value averageis present in the determination range having the non-emission signalvalue average and the signal value average prediction value at the timeof the emission of the auxiliary light source as both ends.

It is preferable that the priority auxiliary light source selecting unitincludes a spatial frequency calculating unit and a priority auxiliarylight source determining unit. The spatial frequency calculating unitthat calculates a spatial frequency of the auxiliary light irradiationareas using each of the auxiliary light sources on the non-emissionimage. The priority auxiliary light source determining unit thatexcludes the auxiliary light source corresponding to the auxiliary lightirradiation areas whose spatial frequency is equal to or smaller than apredetermined value from a selection target of the priority auxiliarylight source and determines that the auxiliary light source remainingafter the excluding is the priority auxiliary light source, in a casewhere the spatial frequency of the auxiliary light irradiation areausing each of the auxiliary light sources is equal to or smaller thanthe predetermined value.

It is preferable that the white balance adjustment value calculatingunit calculates actual-emission priority auxiliary light signalprediction values predicted as the signal values of the priorityauxiliary light irradiation areas at the time of emitting the priorityauxiliary light source with an emission amount at the time of actualemission and calculates a white balance adjustment value based on theactual-emission priority auxiliary light signal prediction values andthe signal values at the time of the non-emission of the priorityauxiliary light irradiation areas.

It is preferable that the white balance adjusting unit obtains an actualemission image obtained by imaging the subject in a state in which theplurality of auxiliary light sources emits light with an emission amountat the time of actual emission and performs the white balance adjustmentusing the white balance adjustment value on the actual emission image.

An operation method of a white balance adjusting apparatus of thepresent invention comprises a non-emission image obtaining step, anemission image obtaining step, an auxiliary light irradiation areaspecifying step, a priority auxiliary light source selecting step, awhite balance adjustment value calculating step, and a white balanceadjusting step. A non-transitory computer readable medium for storing acomputer-executable program for execution of white balance adjustment ofthe present invention causes the computer to perform the above steps. Inthe non-emission image obtaining step, a non-emission image is obtainedby imaging a subject in a state in which a plurality of auxiliary lightsources does not emit light. In the emission image obtaining step,emission images of the auxiliary light sources are obtained by imagingthe subject in a state in which the plurality of auxiliary light sourcesindividually emits light. In the auxiliary light irradiation areaspecifying step, the non-emission image and each of the emission imagesare divided into a plurality of division areas, and auxiliary lightirradiation areas irradiated with auxiliary light of each of theauxiliary light sources are specified based on a signal value differenceof each division area between the state in which the plurality ofauxiliary light sources individually emits light and the state in whichthe plurality of auxiliary light sources does not emit light. In thepriority auxiliary light source selecting step, a priority auxiliarylight source as a target of white balance adjustment is selected fromthe auxiliary light sources. In the white balance adjustment valuecalculating step, a white balance adjustment value is calculated basedon signal values of priority auxiliary light irradiation areasirradiated with auxiliary light of the selected priority auxiliary lightsource. In the white balance adjusting step, adjustment using the whitebalance adjustment value is performed.

According to the present invention, it is possible to provide a whitebalance adjusting apparatus, an operation method thereof, and anon-transitory computer readable medium which allow a main subject tohave an appropriate tint at the time of imaging using a plurality ofauxiliary light sources.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view showing the overall imaging system to whichan embodiment of a white balance adjusting apparatus of the presentinvention is applied, and shows a state in which a pre-emission image iscaptured by turning on a flash light emitting unit of a camera.

FIG. 2 is a functional block diagram of the camera and a flash device.

FIG. 3 is a functional block diagram of a main controller and a digitalsignal processing unit.

FIG. 4 is a flowchart showing WB adjustment in imaging using a pluralityof flash devices.

FIG. 5 is an explanatory diagram showing the specification of flashlight irradiation areas.

FIG. 6 is an explanatory diagram showing the selection of a priorityflash device.

FIG. 7 is an overall perspective view showing a state in which thepre-emission image is captured by turning on a second flash device.

FIG. 8 is a side view showing a flash device including a special effectfilter according to a second embodiment.

FIG. 9 is a functional block diagram showing a priority flash deviceselecting unit of the second embodiment.

FIG. 10 is a flowchart showing WB adjustment according to the secondembodiment.

FIG. 11 is a diagram showing light source color information of ambientlight, light source color information of flash light, and a differencevector in a color space having R/G and B/G on a coordinate axis.

FIG. 12 is a diagram showing a signal value average at the time ofnon-emission of each flash light irradiation areas and a signal valueaverage prediction value at the time of performing irradiation usingflash light in a state in which there is no special effect filter in thecolor space having R/G and B/G on the coordinate axis.

FIG. 13 is a diagram showing the determination of whether or not theflash device is a flash device to which the special effect filter isattached based on whether or not a signal value average at the time ofpre-emission is present in a determination range H1 in the color spacehaving R/G and B/G on the coordinate axis.

FIG. 14 is a diagram showing a determination range H2 according toModification Example 1.

FIG. 15 is a diagram showing a determination range H3 according toModification Example 2.

FIG. 16 is a diagram showing a determination range H4 according toModification Example 3.

FIG. 17 is a diagram showing a determination range H5 according toModification Example 4.

FIG. 18 is a diagram showing a determination range H6 according toModification Example 5.

FIG. 19 is a functional block diagram showing a priority flash deviceselecting unit according to a third embodiment.

FIG. 20 is a flowchart showing WB adjustment according to the thirdembodiment.

FIG. 21 is a functional block diagram showing a priority flash deviceselecting unit according to a fourth embodiment.

FIG. 22 is a flowchart showing WB adjustment according to the fourthembodiment.

FIG. 23 is a front view showing a priority flash device specifyingscreen in a selection input step of the priority flash device accordingto the fourth embodiment.

FIG. 24 is a front view showing a priority flash device determiningscreen in a priority flash device determining step according to thefourth embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS First Embodiment

FIG. 1 is an overall configuration diagram of an imaging system 10 towhich an embodiment of a white balance (hereinafter, referred to as WB)adjusting apparatus of the invention is applied. For example, theimaging system 10 is used in an imaging studio 9 by using a plurality offlash devices 12 and 13 as auxiliary light sources. The imaging system10 includes a digital camera (hereinafter, simply referred to as acamera) 11, and flash devices 12 and 13. The flash device 12 including aflash light emitting unit 14 (see FIG. 2) is built in the camera 11. Thebuilt-in flash device 12 functions as a first auxiliary light source inthe imaging system 10. The flash device 13 is provided separately fromthe camera 11, and functions as a second auxiliary light source in theimaging system 10.

In the imaging system 10, when multi-illumination imaging is performed,the camera 11 controls a turning-on timing by transmitting a controlsignal to the first auxiliary light source (first flash device 12) andthe second auxiliary light source (second flash device 13). The firstflash device 12 irradiates a main subject 6 among subjects 5 with flashlight, and the second flash device 13 irradiates a backdrop 7 disposedbehind the main subject 6 among the subjects 5 with flash light.Although it has been described in the present embodiment that the flashdevice 12 built in the camera 11 is used as the first auxiliary lightsource, a flash device provided separately from the camera 11 or a flashdevice provided integrally with the camera 11 so as to be detachablyattached may be used similarly to the second auxiliary light source.

As shown in FIG. 2, the camera 11 and the flash device 13 includewireless communication interfaces (I/F) 15 and 16, respectively, andthus, the camera 11 and the flash device 13 can wirelessly communicatewith each other. The camera and the flash device may perform wiredcommunication instead of the wireless communication.

The flash device 13 includes a flash controller 17 and a flash lightemitting unit 18 in addition to the wireless communication I/F 16. Theflash device 13 receives a light amount adjusting signal sent from thecamera 11 through the wireless communication I/F 16. The flashcontroller 17 controls the flash light emitting unit 18 to turn on theflash light emitting unit 18 according to the light amount adjustingsignal. The turning-on of the flash light emitting unit 18 is flashemission of which light emission time has a unit of microseconds. Thesame is true of the flash light emitting unit 14 of the flash device 12of the camera 11.

The camera 11 includes a lens barrel 21, an operation switch 22, and arear display unit 23. The lens barrel 21 is provided on a front surfaceof a camera main body 11 a (see FIG. 1), and holds an imaging opticalsystem 25 and a stop 26.

The operation switch 22 is provided in plural on an upper portion or arear surface of the camera main body 11 a. The operation switch 22receives an input operation for power turning ON and OFF operations, arelease operation, and various settings. The rear display unit 23 isprovided on the rear surface of the camera main body 11 a, and displaysimages or live preview images obtained in various imaging modes and menuscreens for performing various settings. A touch panel 24 is provided ona front surface of the rear display unit 23. The touch panel 24 iscontrolled by a touch panel controller 38, and transmits a commandsignal input through a touch operation to a main controller 29.

A shutter 27 and an imaging element 28 are disposed behind the imagingoptical system 25 and the stop 26 in order along an optical axis LA ofthe imaging optical system 25. For example, the imaging element 28 is acomplementary metal-oxide-semiconductor (CMOS) type image sensor of asingle-plate color imaging type having red, green, and blue (RGB) colorfilters. The imaging element 28 images a subject image formed on animaging surface by the imaging optical system 25, and outputs imagingsignals.

The imaging element 28 includes a noise removing circuit, an autogaincontroller, and a signal processing circuit such as an analog/digital(A/D) conversion circuit (all are not shown). The noise removing circuitperforms a noise removing process on the imaging signals. The autogaincontroller amplifies the level of the imaging signal to an optimumvalue. The A/D amplification circuit converts the imaging signals todigital signals, and outputs the digital signals to the imaging element28.

The imaging element 28, the main controller 29, and the flash controller30 are connected to a bus 33. The flash controller 30 and the flashlight emitting unit 14 constitute the flash device 12 built in thecamera 11. In addition, a memory controller 34, a digital signalprocessing unit 35, a media controller 36, a rear display controller 37,and a touch panel controller 38 are connected to the bus 33.

A transitory storage memory 39 such as a synchronous dynamicrandom-access memory (SDRAM) is connected to the memory controller 34.The memory controller 34 inputs and stores image data which are digitalimaging signals output from the imaging element 28 to the memory 39. Thememory controller 34 outputs the image data stored in the memory 39 tothe digital signal processing unit 35.

The digital signal processing unit 35 performs the known imageprocessing such as matrix calculation, demosaicing, WB adjustment, ycorrection, brightness and color difference conversion, resizing, orcompression on the image data input from the memory 39.

The media controller 36 controls the recording and reading of the imagedata in and from a recording media 40. For example, the recording media40 is a memory card having a flash memory built therein. The mediacontroller 36 records the image data compressed by the digital signalprocessing unit 35 in the recording media 40 in a predetermined fileformat.

The rear display controller 37 controls an image display on the reardisplay unit 23. Specifically, the rear display controller 37 generatesvideo signals conforming to the National Television System Committee(NTSC) standard based on the image data generated by the digital signalprocessing unit 35, and outputs the generated video signals to the reardisplay unit 23.

The main controller 29 controls an imaging process of the camera 11.Specifically, the main controller controls the shutter 27 through ashutter drive unit 41. The main controller controls the driving of theimaging element 28 in synchronization with the operation of the shutter27. The camera 11 can set various imaging modes. The main controller 29can perform imaging in the various imaging modes by controlling an Fnumber of the stop 26 or an exposure time of the shutter 27 according tothe set imaging mode.

In the camera 11 according to the present embodiment, amulti-illumination imaging mode is prepared in addition to the variousnormal imaging modes. The multi-illumination imaging modes are selectedat the time of imaging using a plurality of auxiliary light sources. Inthe multi-illumination imaging mode, a priority flash device which is anauxiliary light source to which a priority is given in the WB adjustmentis specified, and a WB adjustment value is calculated based on priorityflash light (priority auxiliary light) which is flash light of thespecified priority flash device. The WB adjustment is performed on anactual emission signal values obtained through the imaging of an actualemission image which is an image at the time of actual emission by usingthe calculated WB adjustment value. The WB adjustment value may becalculated after the priority flash device is specified or the actualemission signals are obtained.

In order to specify the priority flash device, the main controller 29has a priority flash device selecting function. In a case where themulti-illumination imaging mode is selected, a priority flash deviceselecting process is performed before the actual emission image iscaptured. In the present embodiment, one flash device to which apriority is given is selected from two flash devices 12 and 13 in animaging range of the imaging element 28 in the priority flash deviceselecting process.

As shown in FIG. 3, in the multi-illumination imaging mode, the maincontroller 29 functions as an illumination controller 52, an imageobtaining unit 53, a flash light irradiation area specifying unit(auxiliary light irradiation area specifying unit) 54 as an auxiliarylight irradiation area specifying unit, and a priority flash deviceselecting unit 55 as a priority auxiliary light source selecting unit.These respective units are established by starting an operation program45 stored in a nonvolatile memory (not shown) of the camera 11.Similarly, the digital signal processing unit 35 functions as a WBadjusting unit 56, and performs the WB adjustment based on the WBadjustment value.

The image obtaining unit 53 includes a non-emission image obtaining unit53 a and an emission image obtaining unit 53 b. The priority flashdevice selecting unit 55 includes a face area detecting unit 57 and apriority flash device determining unit 58. The WB adjusting unit 56includes a WB adjustment value calculating unit 59.

FIG. 4 is a flowchart showing the WB adjustment in themulti-illumination imaging mode. Initially, in non-emission signal valueobtaining step S11, non-emission images 60 (see FIG. 5(2)) which areimages of the subjects 5 (see FIG. 1) are captured by the imagingelement 28 and the non-emission image obtaining unit 53 a of the imageobtaining unit 53 in a state in which the flash devices 12 and 13 do notemit light. Non-emission signal values are obtained based on thenon-emission images 60.

In pre-emission signal value obtaining step S12, pre-emission images 61and 62 (see FIG. 5(1)) which are images of the subjects 5 are capturedby the imaging element 28 and the emission image obtaining unit 53 b ina state in which the flash devices 12 and 13 do not individually emitlight (individual emission states, see FIGS. 1 and 7), and emissionsignal values are obtained based on the pre-emission images 61 and 62.In this case, the illumination controller 52 controls the turning-ontimings and light amounts of the flash devices 12 and 13 through theflash controller 30 or the wireless communication I/F 15. The emissionimage obtaining unit 53 b selectively turns on the flash devices 12 and13, and obtains the pre-emission image 61 or 62 which is the image ofthe subjects individually irradiated with the flash light.

FIG. 1 shows a state in which the first flash device 12 is turned on atthe time of imaging in the studio. The first flash device 12 is set soas to irradiate the main subject 6 who stands in front of the backdrop 7with the flash light. In this state, the first pre-emission image 61(see FIG. 5(1)) which is the pre-emission image at the time of emissionof first flash light is captured.

FIG. 7 shows a state in which the second flash device 13 is turned on.The second flash device 13 is set so as to irradiate the backdrop 7which is present on the back of the main subject 6 with second flashlight from above on the right side. In this state, the secondpre-emission image 62 (see FIG. 5(1)) which is the pre-emission image atthe time of emission of second flash light is captured.

In FIG. 4, in flash light irradiation area specifying step S13, flashlight irradiation areas irradiated with the flash light from each flashdevice 12 or 13 are specified by the flash light irradiation areaspecifying unit 54.

FIG. 5 is an explanatory diagram showing a flash light irradiation areaspecifying process of the flash light irradiation area specifying unit54 in flash light irradiation area specifying step S13. In the flashlight irradiation area specifying process, flash light irradiation areaspecifying images 63 and 64 are created by using the non-emission images60 and the pre-emission images 61 and 62.

For example, the non-emission images 60 and the pre-emission images 61and 62 are initially divided into 8×8 rectangular division areas 65. Thedivision areas 65 are obtained by dividing the non-emission images 60and the pre-emission images 61 and 62 so as to have the same sections.The number of sections or the shape of the section are not limited tothe illustrated example, and may be appropriately changed. Subsequently,a difference is obtained for each division area 65 by subtracting abrightness value Y0 of each division area 65 obtained from thenon-emission image 60 from a brightness value Ya of each division area65 obtained from the first pre-emission image 61. In a case where thedifference of each division area is larger than those of the otherdivision areas 65, a set of division areas 65 of which the difference islarge is specified as first flash light irradiation areas 67.

In obtaining the non-emission image 60 and the first pre-emission image61, imaging is obtained with a uniform exposure (with the same exposure)at the time of imaging the images 60 and 61. Alternatively, instead ofusing the uniform exposure, a brightness value of the other one of thenon-emission image 60 and the first pre-emission image 61 in relation toa brightness value of one of the images may be corrected based onexposure differences at the time of imaging the images 60 and 61, andthe exposure differences may be corrected through signal processing.

Similarly, a difference is obtained for each division area 65 based on abrightness value Yb of each division area 65 obtained from the secondpre-emission image 62 of the second flash device 13 and a brightnessvalue Y0 of each division area 65 obtained from the non-emission image60, and a set of division areas 65 of which the difference is largerthan those of the other division areas 65 is specified as second flashlight irradiation areas 68. In this case, pre-processing for uniformlyadjusting the exposures at the time of obtaining both the images 60 and62 or post-processing for correcting the brightness value of the otherone of both the images 60 and 62 in relation to the brightness value ofboth the images based on the exposure differences at the time of imagingboth the images 60 and 62 is also performed.

For example, the brightness values Ya, Yb, and Y0 are obtained bycalculating the brightness values of pixels from the followingbrightness conversion expression by using signal values R, G, and B ofthe pixels within each division area.Y=0.3R+0.6G+0.1B

Subsequently, an average brightness value obtained by averaging thebrightness values of the pixels within each division area calculated bythe aforementioned brightness conversion expression is calculated. Forexample, a value to be used is not limited to the aforementionedbrightness value as long as the value is a value representing thebrightness of each division area, and lightness V in the HSV color spaceor lightness L in the Lab color space may be used.

On the first pre-emission image 61, the main subject 6 is positioned inthe center, and the main subject 6 is mainly irradiated with the flashlight (first flash light) from the first flash device 12. Thus, theflash light irradiation areas (first flash light irradiation areas) 67irradiated with the first flash light are specified as represented ashatched portions on the flash light irradiation area specifying image63.

On the second pre-emission image 62 of the second flash device 13, theflash light irradiation areas (second flash light irradiation areas) 68using the second flash device 13 are also specified similarly to thespecification of the first flash light irradiation areas 67. On thesecond pre-emission image 62, since the backdrop 7 is irradiated withthe second flash light as shown in FIG. 7, the second flash lightirradiation areas 68 are specified as represented as the hatchedportions on the flash light irradiation area specifying image 64.

In FIG. 4, in priority flash device selecting step S14, the priorityflash device of the flash devices 12 and 13 is selected as a target ofthe WB adjustment by the priority flash device selecting unit 55.Priority flash device selecting step S14 includes face area detectingstep S15 using the face area detecting unit 57 and priority flash devicedetermining step S16 using the priority flash device determining unit58.

FIG. 6 is an explanatory diagram showing the priority flash deviceselecting process of the priority flash device selecting unit 55 inpriority flash device selecting step S14. Initially, the face areadetecting unit 57 detects face areas 71 of a person from the firstpre-emission image 61 as shown in FIG. 6(5) (face area detecting stepS15). It is preferable that the division areas (division areas smallerthan the division areas 65 by increasing the number of divisions) havingthe size smaller than that of the division areas 65 used at the time ofobtaining the flash light irradiation areas 67 and 68 are used in thedetection of the face areas 71. The face areas 71 may be detected fromthe non-emission image 60 or the second pre-emission image 62.

The priority flash device determining unit 58 specifies which of theflash light irradiation areas 67 and 68 the face areas 71 detected bythe face area detecting unit 57 are present, and determines the flashlight irradiation areas in which the face areas 71 are present aspriority flash light irradiation areas. More specifically, the priorityflash device determining unit obtains information of which of the firstflash light irradiation areas 67 and the second flash light irradiationareas 68 the detected face areas 71 are present from coordinatesrepresenting the mutual positions of these areas on the image. In a casewhere the face areas 71 are present in the first flash light irradiationareas 67, the first flash device 12 which is an emission source of thefirst flash light is automatically specified as the flash device towhich the priority is to be given in the WB adjustment (priority flashdevice determining step S16).

The face areas 71 are detected based on areas indicating the flesh colorof the person. In addition, the face areas 71 may be detected by amethod using shape recognition of eyes, nose, and mouth, a method usingcombination of the flesh color area and the shape recognition, orvarious face recognition methods.

In the present embodiment, it is possible to specify the priority flashdevice by automatically detecting the face areas 71.

A subject image 69 shown in FIG. 6(4) is displayed such that the densityof hatching varies depending on the average brightness values of theflash light irradiation areas 67 and 68, for example, the higher thebrightness values, the higher the density of hatching. Through such adisplay, it is determined for the subject image 69 that the brightnessof the second flash light irradiation areas 68 including mainly thebackdrop 7 is higher than that of the first flash light irradiationareas 67 including the main subject 6. In a case where the brightness ofthe second flash light irradiation areas 68 is high as stated above, theWB adjustment is performed based on the pixels of the second flash lightirradiation areas 68 of which the brightness is high in an automatic WBprocess in the multi-illumination imaging mode of the related art.Accordingly, since the WB adjustment is performed based on the pixels ofthe backdrop 7, the main subject 6 is shifted from an original tint.

In contrast, in the first embodiment, the first flash device 12 isselected as the priority flash device by automatically detecting themain subject 6 as shown in FIG. 6(5) in priority flash devicedetermining step S16. The WB adjustment value is obtained based on thefirst flash light irradiation areas 67 including the main subject 6. TheWB adjustment is performed by using the WB adjustment value, and thus,the main subject 6 can have an appropriate tint.

The WB process is performed in the WB adjusting unit 56 of the digitalsignal processing unit 35. The WB adjustment value calculating unit 59of the WB adjusting unit 56 obtains the WB adjustment value byperforming WB adjustment value calculating step S17 shown in FIG. 4.

WB adjustment value calculating step S17 is performed as follows.Initially, it is assumed that a distribution of brightness values of(i×j) number of divided blocks (division areas 65, i and j=1 to 8 in thepresent example) at the time of emitting only the priority flash lightis Ypre(i, j) and a distribution of brightness values at the time of thenon-emission of the flash light which is only the ambient light is Y0(i,j). Here, a distribution ΔYpre(i, j) of brightness values increased bythe priority flash light is obtained by the following expression.ΔYpre(i,j)=Ypre(i,j)−Y0(i,j)

The imaging is performed at the time of actual emission for performingactual imaging by emitting light with an emission amount which is ktimes an emission amount at the time of pre-emission which is individualemission for obtaining the flash light irradiation areas. K times aredetermined by the dimming result of the camera or the setting of theuser. In this case, in a case where brightness values are increased byonly the priority flash light at the time of actual emission, adistribution ΔYpre(i, j) of brightness values to be expected is obtainedby the following expression.ΔYexp(i,j)−K×ΔYpre(i,j)

In a case where it is assumed that the main subject is irradiated withonly the priority flash light at the time of the actual emission of thelight with the emission amount which is K times the emission amount atthe time of pre-emission, a distribution (actual-emission priorityauxiliary light signal prediction value) Yexp(i, j) of brightness valuesto be expected is obtained by the following expression. Although themain subject is actually irradiated with the other flash, since theinfluence of the other flash is less, the irradiation with the otherflash is excluded.Yexp(i,j)=ΔYexp(i,j)+Y0(i,j)=K×ΔYpre(i,j)+Y0(i,j)

In a case where it is assumed that representative values obtained bycalculating values within the areas irradiated with the priority flashlight through a process such as averaging in the distributions Yexp(i,j) and Y0(i, j) of the brightness values are Yexp#type and Y0#type, αindicating a ratio of the brightness values using the flash light to thebrightness values in the areas irradiated with the priority flash lightis obtained by the following expression.α=(Yexp#type−Y0#type)/Yexp#type

Y0#type corresponds to a signal value at the time of the non-emission ofthe priority auxiliary light irradiation areas.

In a case where it is assumed that the WB adjustment value of theambient light is G0 and the WB adjustment value at the time of emittingonly the flash light recorded within the camera, a WB adjustment valueGwb to be obtained is obtained by the following expression.Gwb=(Gfl−G0)×α+G0

At the time of actual emission, the subjects 5 are captured in a statein which both the first flash device 12 and the second flash device 13emit light, and thus, the actual emission image is obtained. The WBadjusting unit 56 performs WB adjusting step S18 as shown in FIG. 4, andadjusts WB by multiplying the signal values R, G, and B of the actualemission image by the WB adjustment value Gwb. Accordingly, a lightsource color is canceled. The WB adjustment value Gwb is not limited tothe aforementioned method, and may be obtained by various methods.

In the present embodiment, since the priority flash device isautomatically determined based on the main subject 6 and the WBadjustment is performed based on the priority flash device, the mainsubject 6 can have the appropriate tint at the time of imaging using aplurality of flash light rays.

Although it has been described in the aforementioned embodiment that onepriority flash device is used, the WB adjustment value Gwb is obtainedas follows in a case where a plurality of flash devices determined asthe priority flash devices is used.

For example, in a case where two priority flash devices are used and itis initially assumed that distributions of brightness values of (i×j)number of divided blocks at the time of individually emitting firstpriority flash light and second priority flash light are Ypre1(i, j) andYpre2(k, j) and a distribution of brightness values at the time ofnon-emission (=only the ambient light) is Y0(i, j), distributionsΔYpre1(i, j) and ΔYpre2(i, j) of brightness values increased by thefirst and second priority flash light rays are respectively obtained bythe following expressions.ΔYpre1(i,j)=Ypre1(i,j)−Y0(i,j)ΔYpre2(i,j)=Ypre2(i,j)−Y0(i,j)

In a case where the brightness values are increased by only the firstpriority flash light and the second priority flash light at the time ofactual emission, the distribution ΔYexp(i, j) of the brightness valuesto be expected is obtained as follows. K1 is obtained from (emissionamount at the time of actual emission)/(emission amount at the time ofpre-emission) of the first priority flash light, and K2 is obtained from(emission amount at the time of actual emission)/(emission amount at thetime of pre-emission) of the second priority flash light.ΔYexp(i,j)=K1×ΔYpre1(i,j)+K2×ΔYpre2(i,j)

Similarly to the case where one priority flash device is used, thedistributions of the brightness values Yexp(i, j) and Y0(i, j) to beexpected, the representative values of the areas irradiated with thepriority flash light Yexp#type and Y0#type, α indicating a ratio of thebrightness values using the priority flash light to the brightnessvalues in the priority flash light irradiation areas, and so on arecalculated based on the distribution ΔYexp(i, j) of the brightnessvalues corresponding to the obtained increase amount, and the WBadjustment value Gwb is ultimately obtained. The WB adjustment isperformed based on the WB adjustment value Gwb as described above.

Since the face areas are detected from the non-emission image or theemission image by the face area detecting unit 57 and the flash devicecorresponding to the flash light irradiation areas including the faceareas is determined as the priority flash device, the face of the personwhich is the main subject 6 can have the appropriate tint.

Second Embodiment

As shown in FIG. 8, there are some cases where imaging is performed inthe studio by attaching a special effect filter 80 to an irradiationsurface of the flash device 13 and projecting a color or a pattern onthe background. There are many cases where commemorative images arecaptured on the seasons of the year and occasions, and the specialeffect filter 80 is used such that a background color varies dependingon each season of the year or each occasion. For example, in a casewhere an image commemorative of entrance into a school is captured in aschool entrance ceremony on April, the special effect filter 80 forgiving an effect such that the background is in pink or the specialeffect filter 80 for giving an effect such that cherry blossoms arescattered is used in order to express cherry blossoms in full bloom. Thepriority flash device in the imaging in the studio using the specialeffect filter 80 may be automatically selected by excluding the flashdevice for background. In the following embodiments, the same componentsas those of the first embodiment will be assigned the same references,and the redundant description thereof will be omitted.

In the second embodiment, a priority flash device selecting unit 81includes an ambient light coordinate calculating unit 82 that calculatesthe coordinates of the ambient light, a flash light recording unit 83, adifference vector calculating unit 84, a non-emission signal valueaverage calculating unit 85 that calculates the average of the signalvalues at the time of the non-emission of the flash light irradiationareas, a pre-emission signal value average calculating unit 86 thatcalculates the average of the signal values at the time of thepre-emission of the flash light irradiation areas, a signal valueaverage prediction value calculating unit 87, and a special-effect flashlight determining unit 88, as shown in FIG. 9. The priority flash deviceselecting unit 81 identifies that the flash light is the flash lightusing the special effect filter 80, excludes the flash device that emitsthe flash light using the special effect filter 80 from the priorityflash device, and selects the remaining flash device as the priorityflash device.

FIG. 10 is a flowchart showing a process procedure according to thesecond embodiment. Non-emission signal value obtaining step S11,pre-emission signal value obtaining step S12, flash light irradiationarea specifying step S13, WB adjustment value calculating step S17, andWB adjusting step S18 are the same processes as those of the firstembodiment, and only priority flash device selecting step S21 isdifferent. Priority flash device selecting step S21 includes priorityflash device determining step S22 of determining the priority flashdevice through the determination of the image information.

In priority flash device determining step S22, light source coordinates(R0/G0, B0/G0) at a point A representing light source color informationof the ambient light in a color space having R/G and B/G on a coordinateaxis are calculated based on the signal value of the non-emission imageby the ambient light coordinate calculating unit 82, as shown in FIG.11.

Subsequently, light source coordinates (Rf/Gf, Bf/Gf) at a point Brepresenting the light source color information of the flash light inthe same color space are calculated in advance, and are stored in anonvolatile memory by the flash light recording unit 83. Subsequently, avector C which is a difference therebetween is calculated based on thecoordinates (R0/G0, B0/G0) at the point A and the coordinates (Rf/Gf,Bf/Gf) at the point B by the difference vector calculating unit 84. Thevector C is output to the signal value average prediction valuecalculating unit 87.

Subsequently, signal value averages R1, G1, and B1 (corresponding to thepixel information at the time of the non-emission of the auxiliary lightirradiation areas) at the time of the non-emission of the flash lightirradiation areas are calculated, and coordinates (R1/G1, B1/G1) at apoint D in the color space are calculated by the non-emission signalvalue average calculating unit 85, as shown in FIG. 12. The coordinates(R1/G1, B1/G1) at the point D are output to the signal value averageprediction value calculating unit 87 and the special-effect flash lightdetermining unit 88.

Subsequently, coordinates (R2/G2, B2/G2) at a point E in the color spacewhich indicate prediction values R2, G2, and B2 of the signal valueaverages at the time of performing the irradiation using only the flashlight in a state in which there is no special effect filter 80 and thereis no ambient light in the same flash light irradiation areas arecalculated from the following expression by the signal value averageprediction value calculating unit 87. Here, the prediction values R2,G2, and B2 correspond to the signal value average prediction values atthe time of the emission of the auxiliary light source.(R2/G2,B2/G2)=(R1/G1,B1/G1)+C

Subsequently, signal value averages Rpre, Gpre, and Bpre (correspondingto pixel information based on the emission image) in the flash lightirradiation areas of the pre-emission image are obtained by thepre-emission signal value average calculating unit 86, and coordinates(Rpre/Gpre, Bpre/Gpre) at a point F in the color space which indicatethe signal value averages Rpre, Gpre, and Bpre at the time of thepre-emission are calculated as shown in FIG. 13. The coordinates(Rpre/Gpre, Bpre/Gpre) at the point F are output to the special-effectflash light determining unit 88.

Thereafter, the special-effect flash light determining unit 88determines whether or not the flash light is the flash light using thespecial effect filter 80 based on the coordinates (Rpre/Gpre, Bpre/Gpre)at the point F. In a case where the coordinates (Rpre/Gpre, Bpre/Gpre)at the point F are present in a rectangular determination range H1 usingthe point D indicated by the non-emission signal value averagecoordinates (R1/G1, B1/G1) and the point E indicated by theflash-emission signal value average prediction value coordinates (R2/G2,B2/G2) as both ends of a diagonal line, the special-effect flash lightdetermining unit 88 determines that the flash light is the normal flashlight (color temperature: 5000 to 6000K) without using the specialeffect filter 80. In contrast, in a case where the coordinates(Rpre/Gpre, Bpre/Gpre) at the point F are present in the determinationrange H1, it is determined that the flash device is the flash device towhich the special effect filter 80 is attached. Accordingly, in a casewhere the flash device is the flash device to which the special effectfilter 80 is attached, the flash device is excluded from the priorityflash device. Therefore, it is determined that the remaining flashdevice is the priority flash device.

For example, in a case where there is the plurality of flash devicesdetermined as the priority flash devices, it is determined that theflash device having a high brightness value average of the flash lightirradiation areas is the priority flash device. It is determined thatthe flash device having a high light amount set ratio of the user is thepriority flash device. It may be determined that the plurality of flashdevices is the priority flash devices instead of the selecting any onethereof as stated above.

Since the flash device that emits the flash light using the specialeffect filter 80 is excluded from the priority flash device and theremaining flash device is selected as the priority flash device, theflash device that emits the flash light using the special effect filter80 frequently used as the illumination of the background is excludedfrom the priority flash device, and the flash device that outputs theflash light to the main subject 6 such as the person is selected as thepriority flash device. Accordingly, the main subject 6 can have theappropriate tint.

Modification Example 1

Although it has been described in the second embodiment that therectangular determination range H1 is used as shown in FIG. 13, arectangular determination range H2 defined by a width h in a directionperpendicular to a line segment that connects the point D with the pointE is used in Modification Example 1 shown in FIG. 14. For example, as alength which is 30% of a length of the line segment DE is used as thewidth h. Specifically, the width h is set to a value with which WBperformance is the best.

Modification Example 2

In Modification Example 2 shown in FIG. 15, a sector-shaped (fan-shaped)determination range H3 divided up by a predetermined angle θ withrespect to the line segment that connects the point D with the point Ewith the point D as a reference is used. The angle θ is set to a valuewith which the WB performance is the best.

Modification Examples 3 to 5

In Modification Example 3 shown in FIG. 16, a determination range H4obtained by multiplying the vector C by a reduction ratio β (β<1) andfurther reducing the length than the vector C in the determination rangeH1 shown in FIG. 13 is used. Similarly, in Modification Example 4 shownin FIG. 17, a determination range H5 obtained by multiplying the lengthof the line segment DE by the reduction ratio β and further reducing thelength than the line segment DE in the determination range H2 ofModification Example 1 shown in FIG. 14 is used. Similarly, inModification Example 5 shown in FIG. 18, a sector-shaped determinationrange H6 obtained by multiplying the length of the line segment DE bythe reduction ratio β and further reducing the length than the linesegment DE in the determination range H3 of Modification Example 2 shownin FIG. 15 is used.

The reduction ratio β is obtained by the following expression.β=(Ypre−Y0)/Ypre

Ypre is a brightness value average at the time of the pre-emission ofthe flash light irradiation areas, and Y0 is similarly a brightnessvalue average at the time of the non-emission of the flash lightirradiation areas. For example, it is preferable that a margin is givento the reduction ratio β by using a value β1 (=β×1.2) obtained bymultiplying β by 1.2.

As in Modification Examples 1 to 5, it is possible to more strictlydetermine whether or not the flash device is the flash device to whichthe special effect filter 80 is attached by obtaining the determinationranges H2 to H6 other than the determination range H1 shown in FIG. 13.

Although it has been described in the second embodiment that the flashdevice is determined as the priority flash device in a case where theemission signal value average is present in the range including anon-emission signal value average and a flash-light-emission signalvalue average prediction value as both ends, the second embodiment isnot limited to this determination method. For example, the priorityflash device may be determined based on the previously stored pixelinformation of the flash light.

Third Embodiment

As shown in FIG. 19, in a third embodiment, a priority flash deviceselecting unit 90 as a priority auxiliary light source selecting unitincludes a spatial frequency calculating unit 91 and a priority flashdevice determining unit 92, and the priority flash device determiningunit 92 determines whether or not the flash device is the flash devicethat irradiates the background with the flash light based on the spatialfrequency calculated by the spatial frequency calculating unit 91.

FIG. 20 is a flowchart showing a process procedure according to a thirdembodiment. Non-emission signal value obtaining step S11, pre-emissionsignal value obtaining step S12, flash light irradiation area specifyingstep S13, WB adjustment value calculating step S17, and WB adjustingstep S18 are the same processes as those of the first embodiment, andonly priority flash device selecting step S31 is different.

In priority flash device selecting step S31, spatial frequencycalculating step S32 is performed by the spatial frequency calculatingunit 91. In spatial frequency calculating step S32, spatial frequenciesof the flash light irradiation areas 67 and 68 on the non-emissionimages 60 using the flash devices 12 and 13 are calculated.Subsequently, priority flash device determining step S33 is performed bythe priority flash device determining unit 92. In priority flash devicedetermining step S33, in a case where the calculated spatial frequenciesof the flash light irradiation areas 67 and 68 using the flash devices12 and 13 are equal to or smaller than a predetermined value, the flashdevice corresponding to the flash light irradiation areas having thespatial frequency equal to or smaller than the predetermined value isexcluded from a selection target of the priority flash device. There aremany cases where the backdrop 7 is a plain screen, and there are somecases where the spatial frequency is equal to or smaller than thepredetermined value. Accordingly, in the present example, the flashdevice 13 corresponding to the flash light irradiation areas 68 radiatedto the backdrop 7 is excluded, and the flash device 12 corresponding tothe flash light irradiation areas 67 remaining after the excluding isdetermined as the priority flash device. Accordingly, the flash device12 is selected as the priority flash device. In a case where there isthe plurality of flash devices remaining after the excluding, the flashdevice having a high brightness value average in the flash lightirradiation areas is determined as the priority flash device. All theplurality of remaining flash devices may be determined as the priorityflash devices instead of determining only one priority flash device.

Since the flash device corresponding to the flash light irradiationareas in which the spatial frequency is equal to or smaller than thepredetermined value is excluded from the selection target of thepriority flash device and the flash device remaining after the excludingis determined as the priority flash device, the flash device thatirradiates the backdrop 7 is reliably excluded from the selection targetof the priority flash device, and the flash device that irradiates themain subject 6 is selected as the priority flash device. Accordingly,the main subject 6 can have the appropriate tint.

Fourth Embodiment

Although it has been described in the aforementioned embodiments thatthe priority flash device is automatically specified, a manual settingmode in which the priority flash device is manually set is able to beselected and the priority flash device is selected by the user in afourth embodiment shown in FIGS. 21 to 24. As shown in FIG. 21, in themanual setting mode, the rear display unit 23 is controlled through therear display controller 37 by a priority flash device selecting unit 93of the main controller 29.

FIG. 22 is a flowchart showing a process procedure according to a fourthembodiment. Non-emission signal value obtaining step S11, WB adjustmentvalue calculating step S17, and WB adjusting step S18 are the sameprocesses as those of the first embodiment, and only priority flashdevice selecting step S41 is different. Priority flash device selectingstep S41 includes priority flash device selection input step S42,priority flash device determining step S43, priority-flash-lightemission image obtaining step S44, and priority flash light irradiationarea specifying step S45.

In priority flash device selection input step S42, a priority flashdevice specifying screen 94 is displayed on the rear display unit 23 asshown in FIG. 23. A screen title 94 a, selection buttons 94 b and 94 cof the flash devices 12 and 13, and an information text 94 d aredisplayed on the priority flash device specifying screen 94. Thepriority flash device is selected from the selection button 94 b or 94c. The selection is performed by using the touch panel 24. For example,in a case where the flash device 12 of the flash devices 12 and 13 isselected as the priority flash device, the priority flash device isselected by touching the selection button 94 b with a finger 95. Thatis, the touch panel 24 corresponds to a selection input unit that inputsa command to select the priority flash device.

In a case where the selection button 94 b or 94 c, a priority flashdevice determining screen 96 shown in FIG. 24 is displayed on the reardisplay unit 23, and priority flash device determining step S43 isperformed. A title 96 a, the selection buttons 94 b and 94 c,confirmation buttons 96 b and 96 c, or an information text 96 d aredisplayed on the priority flash device determining screen 96. In orderto display that the flash device is selected on the priority flashdevice determining screen 96, the selected selection button (selectionbutton 94 b in FIG. 24) of the selection buttons 94 b and 94 c isdisplayed so as to be turned on and off. In this state, the confirmationbutton 96 b of the confirmation buttons 96 b and 96 c which indicatesYES is touched with the finger 95, and thus, the selected flash device12 is specified as the priority flash device.

Although it has been described that the priority flash device isselected and specified by using the touch panel 24, the method ofspecifying the priority flash device is not limited thereto. Forexample, the priority flash device may be selected and specified byusing the operation switch 22 or using a sound input.

In a case where the priority flash device is determined, emission imageobtaining step S44 of the priority flash light is performed as shown inFIG. 22. In step S44, the pre-emission image 61 (see FIG. 5(1)) which isthe image of the subjects 5 is captured by the emission image obtainingunit 53 b in a state in which only the priority flash device (flashdevice 12 in the present example) emits light (see FIG. 1), and theemission signal values are obtained based on the pre-emission image 61.

In priority flash light irradiation area specifying step S45, the flashlight irradiation area specifying image 63 is created by using thenon-emission image 60 and the pre-emission image 61 as shown in FIG. 5.Initially, the non-emission image 60 and the pre-emission image 61 aredivided into, for example, 8×8 rectangular division areas 65.Subsequently, a difference is obtained for each division area 65 bysubtracting a brightness value Y0 of each division area 65 obtained fromthe non-emission image 60 from a brightness value Ya of each divisionarea 65 obtained from the first pre-emission image 61. In a case wherethe difference is larger than those of the other division areas 65, theset of division areas 65 of which the difference is large is specifiedas the first flash light irradiation areas (priority flash lightirradiation areas) 67.

Thereafter, WB adjustment value calculating step S17 is performed basedon the signal values of the priority flash light irradiation areas, andthe WB is adjusted by using the WB adjustment value in WB adjusting stepS18.

Since the user selects the priority flash device, it is possible tosimply determine the priority flash device without performing acomplicated process such as the detection of the face area, thecalculation of the signal value average, or the calculation of thespatial frequency.

In the embodiments, the hardware structure of the processing units thatperform various processing such as the non-emission image obtaining unit53 a, the emission image obtaining unit 53 b, the flash lightirradiation area specifying unit (auxiliary light irradiation areaspecifying unit) 54, the priority flash device selecting unit (priorityauxiliary light source selecting unit) 55, 81, 90, or 93, the WBadjustment value calculating unit 59, the WB adjusting unit 56, the facearea detecting unit 57, the priority flash device determining unit(priority auxiliary light source determining unit) 58 or 92, and thespatial frequency calculating unit 91 is realized by various processorsas follows. The various processors include a central processing unit(CPU) which is a general-purpose processor functioning as variousprocessing units, a programmable logic device (PLD) which is a processorcapable of changing a circuit configuration after a field-programmablegate array is manufactured, and a dedicated electric circuit which is aprocessor having a dedicated circuit configuration designed forperforming a specific process such as an Application-Specific IntegratedCircuit (ASIC).

On processing unit may be constituted by one of the various processors,or may be constituted by a combination (for example, a combination of aplurality of FPGAs or a combination of the CPU and the FPGA) of two orprocessors of the same type or different types. The plurality ofprocessing units may be constituted by one processor. An example inwhich the plurality of processing units is constituted by one processoris as follows. Firstly, one processor is constituted by a combination ofone or more CPUs and software, and this processor functions as theplurality of processing units. Secondly, a processor that realizes allthe functions of the system including the plurality of processing unitsby using one integrated circuit (IC) chip, such as system on chip (SoC),is used. As stated above, the various processing units are constitutedby one or more processors of the various processors as a hardwarestructure.

More specifically, the hardware structure of the various processors isan electric circuitry obtained by combining circuit elements such assemiconductor elements.

From the above description, it is possible to ascertain the inventionrepresented by the following appendix.

APPENDIX 1

There is provided a white balance adjusting apparatus including anon-emission image obtaining processor that obtains a non-emission imageby imaging a subject in a state in which a plurality of auxiliary lightsources does not emit light, an emission image obtaining processor thatobtains emission images of the auxiliary light sources by imaging thesubject in a state in which the plurality of auxiliary light sourcesindividually emits light, an auxiliary light irradiation area specifyingprocessor that divides the non-emission image and each of the emissionimages into a plurality of division areas, and specifies auxiliary lightirradiation areas irradiated with auxiliary light of each of theauxiliary light sources based on a signal value difference of eachdivision area between the state in which the plurality of auxiliarylight sources individually emits light and the state in which theplurality of auxiliary light sources does not emit light, a priorityauxiliary light source selecting processor that selects a priorityauxiliary light source as a target of white balance adjustment from theauxiliary light sources, a white balance adjustment value calculatingprocessor that calculates a white balance adjustment value based onsignal values of priority auxiliary light irradiation areas irradiatedwith auxiliary light of the selected priority auxiliary light source,and a white balance adjusting processor that performs adjustment usingthe white balance adjustment value.

The present invention is not limited to the embodiments or themodification examples, and may adopt various configurations withoutdeparting from the gist of the present invention. For example, theembodiments or the modification examples may be appropriately combined.

The present invention is applicable to an imaging device such as amobile phone or a smartphone in addition to the camera 11.

EXPLANATION OF REFERENCES

-   -   5: subject    -   6: main subject    -   7: backdrop    -   9: imaging studio    -   10: imaging system    -   11: digital camera (camera)    -   11 a: camera main body    -   12: first flash device (auxiliary light source)    -   13: second flash device (auxiliary light source)    -   14: flash light emitting unit    -   15, 16: wireless communication I/F    -   17: flash controller    -   18: flash light emitting unit    -   21: lens barrel    -   22: operation switch    -   23: rear display unit    -   24: touch panel    -   25: imaging optical system    -   26: stop    -   27: shutter    -   28: imaging element    -   29: main controller    -   30: flash controller    -   33: bus    -   34: memory controller    -   35: digital signal processing unit    -   36: media controller    -   37: rear display controller    -   38: touch panel controller    -   39: memory    -   40: recording media    -   41: shutter driver unit    -   45: operation program    -   52: illumination controller    -   53: image obtaining unit    -   53 a: non-emission image obtaining unit    -   53 b: emission image obtaining unit    -   54: flash light irradiation area specifying unit    -   55: priority flash device selecting unit (priority auxiliary        light source selecting unit)    -   56: WB adjusting unit (white balance adjusting unit)    -   57: face area detecting unit    -   58: priority flash device determining unit (priority auxiliary        light source determining unit)    -   59: WB adjustment value calculating unit (white balance        adjustment value calculating unit)    -   60: non-emission image    -   61, 62: first and second pre-emission images    -   63, 64: flash light irradiation area specifying image    -   65: division area    -   67: first flash light irradiation area    -   68: second flash light irradiation area    -   69: subject image    -   71: face area    -   80: special effect filter    -   81: priority flash device selecting unit (priority auxiliary        light source selecting unit)    -   82: ambient light coordinate calculating unit    -   83: flash light recording unit    -   84: difference vector calculating unit    -   85: non-emission signal value average calculating unit    -   86: pre-emission signal value average calculating unit    -   87: signal value average prediction value calculating unit    -   88: special-effect flash light determining unit    -   90: priority flash device selecting unit (priority auxiliary        light source selecting unit)    -   91: spatial frequency calculating unit    -   92: priority flash device determining unit (priority auxiliary        light source determining unit)    -   93: priority flash device selecting unit (priority auxiliary        light source selecting unit)    -   94: priority flash device specifying screen    -   94 a: title    -   94 b, 94 c: selection button    -   94 d: information text    -   95: finger    -   96: priority flash device determining screen    -   96 a: title    -   96 b, 96 c: confirmation button    -   96 d: information text    -   A: light source coordinates of ambient light source    -   B: light source coordinates of flash light    -   C: vector    -   D: non-emission signal value average of flash light irradiation        area    -   DE: line segment    -   E: signal value average prediction value at the time of emitting        only flash of flash light irradiation area    -   H1 to H6: determination range    -   LA: optical axis    -   S11: non-emission signal value obtaining step    -   S12: pre-emission signal value obtaining step    -   S13: flash light irradiation area specifying step (auxiliary        light irradiation area specifying step)    -   S14: priority flash device selecting step (priority auxiliary        light source selecting step)    -   S15: face area detecting step    -   S16: priority flash device determining step    -   S17: WB adjustment value calculating step (white balance        adjustment value calculating step)    -   S18: WB adjusting step (white balance adjusting step)    -   S21, S31, S41: priority flash device selecting step (priority        auxiliary light source selecting step)    -   S22: priority flash device determining step    -   S32: spatial frequency calculating step    -   S33: priority flash device determining step    -   S42: priority flash device selection input step    -   S43: priority flash device determining step    -   S44: priority flash light emission image obtaining step    -   S45: priority flash light irradiation area specifying step    -   h: width    -   β: reduction ratio    -   θ: angle

What is claimed is:
 1. A white balance adjusting apparatus comprising: aprocessor that: obtains a non-emission image by imaging a subject in astate in which a plurality of auxiliary light sources does not emitlight; obtains emission images of the auxiliary light sources by imagingthe subject in a state in which the plurality of auxiliary light sourcesindividually emits light; divides the non-emission image and each of theemission images into a plurality of division areas, and specifiesauxiliary light irradiation areas irradiated with auxiliary light ofeach of the auxiliary light sources based on a signal value differenceof each division area between the state in which the plurality ofauxiliary light sources individually emits light and the state in whichthe plurality of auxiliary light sources does not emit light; selects apriority auxiliary light source as a target of white balance adjustmentfrom the auxiliary light sources; calculates a white balance adjustmentvalue based on signal values of priority auxiliary light irradiationareas irradiated with auxiliary light of the selected priority auxiliarylight source; and performs adjustment using the white balance adjustmentvalue.
 2. The white balance adjusting apparatus according to claim 1,wherein the processor further: detects face areas from the non-emissionimage or the emission images, and specifies which of the auxiliary lightirradiation areas the face areas detected are present, and determinesthat the auxiliary light source corresponding to the auxiliary lightirradiation areas including the face areas is the priority auxiliarylight source.
 3. The white balance adjusting apparatus according toclaim 1, wherein the processor further determines the priority auxiliarylight source based on previously stored light source color informationof the auxiliary light source.
 4. The white balance adjusting apparatusaccording to claim 3, wherein the processor further sets a determinationrange in a color space by using the previously stored light source colorinformation of the auxiliary light, light source color information ofambient light obtained from the non-emission image, and pixelinformation at the time of non-emission of the auxiliary lightirradiation areas, and determines the auxiliary light sourcecorresponding to the auxiliary light irradiation areas as the priorityauxiliary light source in a case where the pixel information based onthe emission image is positioned within the determination range.
 5. Thewhite balance adjusting apparatus according to claim 4, wherein thelight source color information of the auxiliary light is coordinatesindicating a color of the auxiliary light in a color space, the lightsource color information of the ambient light is coordinates which areobtained based on the non-emission image and indicate a color of theambient light in the color space, the pixel information at the time ofthe non-emission of the auxiliary light irradiation areas is coordinateswhich are obtained based on the non-emission image and indicate anon-emission signal value average of the auxiliary light irradiationareas in the color space, and the processor further calculates adifference vector which is a difference between the coordinates of theauxiliary light and the coordinates of the ambient light, obtains asignal value average prediction value at the time of emission of theauxiliary light source by adding the difference vector to thecoordinates of the non-emission signal value average, calculates anemission signal value average which is a signal value average of theauxiliary light irradiation areas in the color space based on theemission image, and determines the priority auxiliary light source basedon the non-emission signal value average, a signal value averageprediction value at the time of the emission of the auxiliary lightsource, and the emission signal value average.
 6. The white balanceadjusting apparatus according to claim 5, wherein, in a case where theemission signal value average is present in the determination rangehaving the non-emission signal value average and the signal valueaverage prediction value at the time of the emission of the auxiliarylight source as both ends, processor further determines that theauxiliary light source corresponding to the auxiliary light irradiationareas is the priority auxiliary light source.
 7. The white balanceadjusting apparatus according to claim 1, wherein the processor further:calculates a spatial frequency of the auxiliary light irradiation areasusing each of the auxiliary light sources on the non-emission image, andexcludes the auxiliary light source corresponding to the auxiliary lightirradiation areas whose spatial frequency is equal to or smaller than apredetermined value from a selection target of the priority auxiliarylight source and determines that the auxiliary light source remainingafter the excluding is the priority auxiliary light source, in a casewhere the spatial frequency of the auxiliary light irradiation areausing each of the auxiliary light sources is equal to or smaller thanthe predetermined value.
 8. The white balance adjusting apparatusaccording to claim 1, wherein the processor further calculatesactual-emission priority auxiliary light signal prediction valuespredicted as the signal values of the priority auxiliary lightirradiation areas at the time of emitting the priority auxiliary lightsource with an emission amount at the time of actual emission, andcalculates a white balance adjustment value based on the actual-emissionpriority auxiliary light signal prediction values and the signal valuesat the time of the non-emission of the priority auxiliary lightirradiation areas.
 9. The white balance adjusting apparatus according toclaim 1, wherein the processor further obtains an actual emission imageobtained by imaging the subject in a state in which the plurality ofauxiliary light sources emits light with an emission amount at the timeof actual emission, and performs the white balance adjustment using thewhite balance adjustment value on the actual emission image.
 10. Anoperation method of a white balance adjusting apparatus, the methodcomprising: a non-emission image obtaining step of obtaining anon-emission image by imaging a subject in a state in which a pluralityof auxiliary light sources does not emit light; an emission imageobtaining step of obtaining emission images of the auxiliary lightsources by imaging the subject in a state in which the plurality ofauxiliary light sources individually emits light; an auxiliary lightirradiation area specifying step of dividing the non-emission image andeach of the emission images into a plurality of division areas, andspecifies auxiliary light irradiation areas irradiated with auxiliarylight of each of the auxiliary light sources based on a signal valuedifference of each division area between the state in which theplurality of auxiliary light sources individually emits light and thestate in which the plurality of auxiliary light sources does not emitlight; a priority auxiliary light source selecting step of selecting apriority auxiliary light source as a target of white balance adjustmentfrom the auxiliary light sources; a white balance adjustment valuecalculating step of calculating a white balance adjustment value basedon signal values of priority auxiliary light irradiation areasirradiated with auxiliary light of the selected priority auxiliary lightsource; and a white balance adjusting step of performing adjustmentusing the white balance adjustment value.
 11. A non-transitory computerreadable medium for storing a computer-executable program for executionof white balance adjustment, the computer-executable program causing acomputer to perform: a non-emission image obtaining step of obtaining anon-emission image by imaging a subject in a state in which a pluralityof auxiliary light sources does not emit light; an emission imageobtaining step of obtaining emission images of the auxiliary lightsources by imaging the subject in a state in which the plurality ofauxiliary light sources individually emits light; an auxiliary lightirradiation area specifying step of dividing the non-emission image andeach of the emission images into a plurality of division areas, andspecifies auxiliary light irradiation areas irradiated with auxiliarylight of each of the auxiliary light sources based on a signal valuedifference of each division area between the state in which theplurality of auxiliary light sources individually emits light and thestate in which the plurality of auxiliary light sources does not emitlight; a priority auxiliary light source selecting step of selecting apriority auxiliary light source as a target of white balance adjustmentfrom the auxiliary light sources; a white balance adjustment valuecalculating step of calculating a white balance adjustment value basedon signal values of priority auxiliary light irradiation areasirradiated with auxiliary light of the selected priority auxiliary lightsource; and a white balance adjusting step of performing adjustmentusing the white balance adjustment value.